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ABSTRACT 


In  the  current  environment  of  military  operations  requesting  faster  delivery 
schedules  to  counter  insurgent  tactics,  the  engineering  team  often  searches  for  how  to 
quickly  deliver  the  “80%  solution”,  typically  in  6-12  months.  These  are  labeled  rapid 
development  projects.  A  content  analysis  of  best  practices  in  commercial  product 
development  literature,  where  time  to  market  is  often  a  driving  factor,  was  accomplished 
showing  varying  emphasis  of  systems  engineering  technical  and  technical  management 
processes.  Technical  Planning,  Stakeholders  Requirements  Development,  and  Architecture 
Design  were  identified  as  important  processes.  This  analysis  confirms  preconceived  notions 
of  “plan  upfront  and  early”  by  emphasizing  the  SE  processes  of  Stakeholder  Requirements 
Definition,  Architecture  Design  and  Technical  Planning.  A  purposive  sampling  of  AFRL 
rapid  development  program  managers  and  engineers  was  conducted  to  identify  important 
SE  processes  and  compared  to  the  literature  content  analysis.  The  results  of  this  sampling 
did  not  strongly  emphasize  one  process  over  another  however  Architecture  Design, 
Implementation  scored  higher  among  Technical  Processes.  Decision  Analysis,  Technical 
Planning,  Technical  Assessment  and  Data  Management  scored  slightly  higher  among 
Technical  Management  Processes.  Anecdotal  evidence  also  emphasized  iterating  prototype 
designs  based  on  early  customer  feedback,  focusing  mostly  on  critical  risks  and  holding 
more  reviews  early  in  a  project  schedule  until  a  trust  in  the  team  is  built. 
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RAPID  DEVELOPMENT:  A  CONTENT  ANALYSIS  COMPARISON  OF 
LITERATURE  AND  PURPOSIVE  SAMPLING  OF  AFRL  RAPID  REACTION 

PROJECTS 


I.  INTRODUCTION 

The  accelerated pace  of  change  in  the  tactics,  techniques  and procedures  used  by  adversaries  of  the 
United  States  has  heightened  the  need  for  a  rapid  response  to  neiv  threats.  Fielding  systems  in  response  to  urgent 
operational  needs  over  the  last  half  decade  has  revealed  the  DoD  lacks  the  ability  to  rapidly  field  new  capabilities 
for  the  wa  fighter  in  a  systematic  and  effective  way.  —  Report  of  the  Defense  Science  Board  Task  Force  on 
Fulfilling  Urgent  Operational  Needs,  July  2009 

Background 

The  Department  of  Defense  (DoD)  acquisition  system  is  chartered  with  providing 
effective,  affordable,  and  timely  systems  to  our  operational  forces  (DoD  5000.01).  From 
Los  Angeles-class  submarines  to  the  M1A1  Abrams  tank  to  the  F-22  Raptor,  the  DoD  has 
produced  the  most  technologically  advanced  weapon  systems  ever  made.  With  a  workforce 
of  130,000  (OSD/AT&L,  2010),  the  acquisition  community  delivers  the  tools  enabling  our 
military  to  perfonn  the  missions  they  are  tasked  to  accomplish. 

The  process  by  which  we  develop  those  warfighting  tools  has  continually  evolved 
to  meet  the  changing  times.  A  RAND  study  of  acquisition  reform  (Hanks  et  al,  2005)  lists 
major  events  in  acquisition  refonn  as  shown  in  Table  1,  to  which  this  author  has  modified 
for  brevity  and  included  recent  revisions  to  the  DoD  5000  series  of  instruction  that  guides 
the  execution  of  programs. 
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Table  1.  Acquisition  Reform  Milestones 


Date 

Major  Acquisition  Event 

1972 

Commission  on  Government  Procurement 

1974 

Office  of  Federal  Procurement  Policy  Act 

1982 

Executive  Order  12352  (established  the  FAR  and  directed  procurement  reforms) 

1983 

Grace  Commission 

1983 

Office  of  Federal  Procurement  Policy  Act 

1984 

Federal  Acquisition  Regulation  (FAR) 

1985 

Department  of  Defense  Procurement  Reform  Act 

1986 

Department  of  Defense  Reorganization  Act  (“Goldwater-Nichols  Act”) 

1986 

Packard  Commission 

1990 

Defense  Acquisition  Workforce  Improvement  Act 

1993 

Acquisition  Law  Advisory  Panel 

1993 

Government  Perfonnance  and  Results  Act 

1994 

Secretary  of  Defense  Perry”s  “Acquisition  Reform:  A  Mandate  for  Change” 

1994 

DUSD  for  Acquisition  Refonn  Office  first  established 

1994 

Federal  Acquisition  Streamlining  Act  (FAS A) 

1995 

Commission  on  Defense  Roles  and  Missions  (CORM) 

1996 

Administrative  Dispute  Resolution  Act 

1996 

Clinger-Cohen  Act 

1997 

Defense  Refonn  Act 

1997 

Quadrennial  Defense  Review  #1,  issued  May  1997  (called  for  by  FY95  NDAA) 

1998 

Acquisition  Results  Act 

2001 

DoD  5000  rewrite 

2007 

DoD  5000.01  Revised 

2008 

DoD  5000.02  Revised 

2009 

Weapon  System  Acquisition  Refonn  Act  (WSARA) 

Currently,  the  United  States  is  challenged  in  responding  to  new  emerging  threats, 
specifically  in  the  proliferation  of  the  improvised  explosive  device  (IED).  To  complicate 
this  threat,  the  enemy  uses  readily  available  commercial  items  and  various  explosive 
materials  to  build  IEDs,  combat  tests  hundreds  of  combinations  of  these  devices  aided  by 
blending  into  the  local  population  and  by  the  covert  nature  of  the  devices,  and 
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communicates  lessons  learned  and  success  stories  across  shadow  websites  on  the  Internet 


(JIEDDO,  2009). 

The  impact  of  our  enemies"  ability  to  produce  weapon  systems  quicker,  cheaper 
and  within  reach  of  our  forces  has  lead  to  numerous  studies  on  how  to  rapidly  field  new 
capabilities  to  the  warfighter  (DSB  2007,  2009;  GAO  2010;  Solomon,  2008).  Anecdotal 
reviews  of  prior  wartime  acquisition  offer  the  insight  that  it  is  possible  to  respond  to 
emerging  threats  in  a  responsive  manner  to  give  our  warfighters  the  advantage.  Radar 
stations  developed  before  and  during  WWII  provided  the  British  and  Americans  early 
warning  for  incoming  Gennan  bombers  (Brown,  1999).  The  Culin  Hedgerow  Cutter  was 
adapted  from  steel  obstacles  (originally  emplaced  by  the  Gennan  anny)  and  attached  to  the 
front  of  Shennan  tanks  allowing  the  breaching  of  hedgerows  to  counter  Gennan 
emplacements  in  confined  fields  in  the  taking  of  the  French  town  of  St.  Lo  (Guttman, 
1998).  Electronic  countermeasures  were  implemented  in  F-100,  F-105  and  F-4  Wild 
Weasel  squadrons  to  locate  and  negate  surface-to-air  (SAM)  site  threats  during  Vietnam 
(Hewitt,  1992).  The  United  States  military  has  a  history  of  quickly  implemented  responses 
to  emerging  threats. 

In  fact,  there  are  cunent  efforts  to  provide  our  warfighters  with  timely  solutions  to 
their  needs.  The  Defense  Science  Board  identified  no  less  than  20  groups  dedicated  to  such 
a  task  (DSB,  2009).  These  organizations  were  found  at  many  levels  from  the  Office  of  the 
Secretary  of  Defense  to  the  Major  Commands  (MAJCOMs)  in  the  services  to  the 
Combatant  Commands  (COCOMs)  themselves.  Some  were  focused  on  a  specific  threat  or 
capability,  like  the  Joint  IED  Defeat  Organization  (JIEDDO)  or  the  Intelligence, 
Surveillance  and  Reconnaissance  (ISR)  and  Mine  Resistant  Ambush  Protected  (MRAP) 
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Task  Forces,  while  others  sought  a  way  to  handle  the  broader  rapid  fielding  process,  like 
the  Rapid  Reaction  Technology  Office  or  the  Anny  Rapid  Equipping  Force. 

Within  the  Air  Force,  the  future  technology  capabilities  are  discovered,  developed 
and  delivered  in  the  Air  Force  Research  Laboratory.  AFRL  has  aligned  its  programs  along 
three  Core  Processes,  each  focused  on  the  different  stages  from  science  studies  to 
technology  insertion.  Core  Process  3  (CP3)  addresses  the  immediate  needs  requested  by  the 
warfighter  and  delivers  a  demonstration  prototype  within  “12  months  or  less”  (AFRL 
Instruction  90-104,  Vol  3).  While  not  fully  matured  along  standard  acquisition 
requirements,  the  prototype  is  expected  to  be  used  in  the  field  upon  completion  of  a 
successful  demonstration.  During  the  development  effort,  transition  partners  identify  paths 
to  insert  the  capability  into  programs  of  record,  if  desired. 

Recently,  AFRL  issued  an  instruction  for  executing  the  CP3  mission.  AFRLI  90- 
104,  Vol  3,  lays  out  general  organization  strategies  such  as  forming  “rapid  reaction  teams” 
and  iNodes  by  matrixing  subject  matter  experts  from  across  AFRL,  industry  and  academia 
to  solve  urgent  needs.  General  guidance  to  meet  timelines  and  frequent  process  owner 
updates  combined  with  organizational  “hard  chargers”  ensure  prototypes  are  delivered  on 
time.  Currently  there  is  no  collection  of  lessons  learned  or  best  practices  that  would  assist  a 
program  manager  in  creating  a  development  strategy  on  short  timeframes. 

Previous  studies  have  investigated  1)  how  DoD  rapid  development/  rapid 
acquisition  organizations  use  innovation  to  meet  urgent  needs  (Behm  et  al,  2009)  and  2) 
how  AFRL  implements  a  systems  engineering  approach  across  all  its  programs  to 
effectively  deliver  products  to  the  acquisition  community  (Solomon,  2008).  This  effort  will 
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synthesize  the  two  ideas  to  identify  the  systems  engineering  practices  necessary  for 
successful  rapid  development  efforts  within  AFRL. 

Problem  Statement  and  Objective 

Complex  weapon  systems  require  a  level  of  organization  to  communicate  designs, 
establish  milestones  and  lay  out  a  schedule.  The  field  of  systems  engineering  has  developed 
a  framework  with  a  track  record  of  helping  programs  stay  on  cost  and  on  time  (Honour, 
2004).  However,  systems  engineering  (SE)  is  perceived  in  the  science  and  technology 
(S&T)  culture  of  AFRL  as  non-value  added  (Behm  et  al,  2009;  Doyle,  2008).  However,  if  a 
traditional  SE  approach  can  be  tailored  and  validated  for  rapid  development  projects,  this 
would  be  an  approach  well  suited  to  meet  user  expectations  by  delivering  quality  products 
along  aggressive  schedules.  The  objective  is  to  develop  such  a  framework  through 
literature  review  and  validate  by  studying  recent  rapid  development  efforts  in  AFRL. 
Research  questions 

1 .  What  accepted  activities  in  rapid  development  literature  and  practice  correlate  to 
Defense  Acquisition  SE  activities? 

2.  What  SE  activities  were  emphasized  by  AFRL  program  managers,  lead  engineers 
and  key  personnel  on  recent  rapid  development  projects? 

3.  How  does  the  model  reflecting  the  literature  compare  to  the  model  found  in  AFRL 
rapid  development  projects? 

Methodology 

A  review  of  literature  will  identify  industry  best  practices  for  rapid  development 
and  systems  engineering.  Out  of  this  review,  a  framework  of  key  practices  for  rapid 
development  will  be  derived  from  a  comparison  of  current  DoD  suggested  practices  for 
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systems  engineering.  A  purposive  sampling  of  AFRL  rapid  reaction  team  members  will 
identify  key  SE  activities  utilized  in  recent  projects  and  will  be  compared  to  the  model 
fonned  by  the  literature  study.  Finally,  a  recommendation  of  best  practices  will  be  crafted 
for  future  AFRL  CP3  projects  conveyed  in  draft  language  for  updates  to  the  current  AFRL 
Instruction  90-104,  Volume  3,  “AFRL  Core  Process  3,  Innovative  Solutions  to  Near-Term 
Needs”. 

Summary 

This  chapter  identified  the  need  for  rapid  development  and  current  challenges  faced 
by  the  DoD.  AFRL  has  instituted  Core  Process  3  to  handle  rapid  development  projects  to 
meet  urgent  needs  of  the  warfighter.  Instituting  best  practices  of  successful  rapid 
development  projects  within  AFRL  identified  by  literature  review  and  validated  with  case 
studies  will  increase  the  success  of  CP3  projects.  Chapter  2  will  provide  a  literature  review 
of  the  DoD"s  acquisition  system,  its  efforts  to  meet  urgent  warfighter  needs,  and  best 
practices  of  rapid  development  approaches  in  the  academic  and  business  literature.  Chapter 
3  will  provide  the  methodology  to  detennine  a  tailored  systems  engineering  approach  for 
CP3  projects  within  AFRL.  Chapter  4  will  compare  the  proposed  framework  with  the  case 
studies  and  present  the  results  and  assess  the  importance  of  SE  activities  in  those  case 
studies.  Chapter  5  will  then  evaluate  the  framework  and  identify  any  possible 
improvements  and  conclude  with  a  tailored  SE  model  for  rapid  development  projects 
conducted  under  AFRL's  Core  Process  3. 
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II.  LITERATURE  REVIEW 


Current  DoD  Acquisition  and  Rapid  Reaction  Efforts 

Formal  Department  of  Defense  acquisition  processes  and  organizations  have  been 
slow  and  unresponsive  to  initial  requests  to  counter  the  IED  threat  (DSB,  2009).  The 
acquisition  model  currently  used  by  the  Department  of  Defense  is  based  on  three  highly 
interrelated  and  complex  processes  to  deliver  weapon  systems  to  our  anned  forces  as 
outlined  in  the  Defense  Acquisition  Guidebook-  see  Figure  1.  The  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  process  identifies  gaps  in  current  warfighter 
capabilities  and  proposes  solutions  to  fill  those  gaps.  The  Planning,  Programming, 
Budgeting  and  Execution  (PPBE)  process  makes  the  monetary  and  programmatic 
investments  based  on  the  prioritized  list  of  gaps  and  solutions  detennined  by  the  JCIDS 
process.  The  Defense  Acquisition  System  executes  programs  based  on  the  funding  they 
receive  to  deliver  a  product  to  the  warfighter. 

The  funding  generally  operates  on  a  two  year  cycle  and  justification  to  the  Joint 
Chiefs  of  Staff  is  generally  required  to  make  any  major  changes  to  the  plan.  The  gaps 
discovered  by  the  JCIDS  process  are  initiated  by  requirements  from  the  operational  user 
who  perceives  a  shortfall  in  capability  of  the  equipment  developed  and  procured  for  them. 
During  wartime,  solutions  are  needed  much  quicker  than  starting  in  the  next  two-year 
cycle.  In  response,  many  ad  hoc  organizations  have  sprung  up  and  established 
organizations  have  developed  new  processes  to  meet  the  thousands  of  requirements,  Joint 
and  Urgent  Operational  Needs  (JUONs  and  UONs)  as  defined  in  CJCS  Instruction 
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3470.01,  coming  from  combatant  commanders.  Figure  2  displays  some  of  these 
organizations. 


Joint  Capabilities 
Integration  & 
Development 
System  (JCIDS) 

VCJCS/JROC 

Oversight 


Defense 

Acquisition 

System 

Milestone  Decision 
Authority  fVDA) 
Oversight 


CJCS  3170.01  Series 


MID  913  PPBSto  PPBE 


DoD  5000  Series 


Figure  1:  DoD  Decision  Support  System  (DAG,  Chi) 

The  Office  of  the  Under  Secretary  of  Defense  has  established  the  Joint  Rapid 
Acquisition  Cell  (JRAC)  to  be  the  focal  point  for  responding  to  JUONs.  To  manage 
COCOM  requests  that  need  a  timely  response,  the  DoD  has  created  a  subset  of  JUONs 
called  Immediate  Warfighter  Needs  (IWNs).  These  requests  are  designated  by  the  JRAC  as 
needing  a  material  or  logistic  solution  within  120  days.  The  JRAC  then  works  with  the 
appropriate  service  or  organization  to  find  a  solution  within  120  days,  which  if  approved  is 
delivered  to  the  COCOM. 
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Dete:  Jwtuiy  240t 


Figure  2:  DoD  Rapid  Reaction  Organizations  (DSB,  Urgent  Needs,  2009) 


The  Air  Force  has  established  its  own  Rapid  Response  Process  as  codified  in  AFI 
63-114.  While  oriented  towards  Air  Force  UONs,  it  has  the  capability  to  respond  to  JUONs 
if  the  solution  resides  within  the  Air  Force  Space  and  Missile  System  Center  (AFSPC),  the 
Air  Force  Materiel  Command  (AFMC),  the  Air  Force  Research  Laboratory  (AFRL)  or  the 
Program  Executive  Officers.  The  process  starts  with  a  COCOM  submitting  a  UON  to  the 
lead  MAJCOM  (ACC,  AMC,  AFSOC,  or  AFSPC)  which  has  the  mission  or  capability 
shortfall  addressed  by  the  UON.  A  Combat  Capability  Document  (CDD)  may  then  be 
delivered  to  Headquarters  Air  Force  for  approval.  This  initiates  the  Rapid  Response 
Process  (RRP)  which  then  reviews  the  CCD  based  on  a  set  of  criteria  that  includes 
timeliness  of  the  solution,  need  of  the  capability  to  address  the  shortfall,  whether  the 
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capability  is  “operationally  safe,  suitable  and  effective,  supportive,  sustainable,  affordable 
with  the  support  infrastructure  already  in  place,”  do  not  require  RDT&E  to  field  and  that 
the  CCD  has  addressed  Mishap  Prevention  per  AFI  91-202.  If  the  CCD  is  approved 
without  issues,  HQAF  is  notified  and  the  solution  is  implemented.  If  the  CCD  does  not 
meet  the  RRP  criteria,  the  lead  MAJCOM  may  submit  their  request  through  the  JCIDS 
process. 

This  emphasis  on  potentially  lengthy  upfront  analysis  and  preparation  has  created  a 
deterrent  to  pursue  the  RRP  and  incentive  to  find  other  means  of  answering  the  J/UONs.  As 
described  in  CJCSI  3470.01  which  addresses  how  JUONs  are  validated  and  funded, 

In  most  cases,  the  lead  MAJCOM  satisfies  the  combatant 
commandefi's  urgent  need  through  means  other  than  the  CCD  process 
(non-materiel  solution,  internal  programming  authority,  off-the-shelf 
purchase,  etc.).  This  is  the  preferred  method  as  it  provides  the  quickest 
support  to  the  warfighter. 

In  addition  to  the  above  instruction,  recent  discussions  with  HQ  AFMC  staff  members 
validated  the  above  statement  by  noting  that  no  CCD  has  been  written  in  the  past  year  for 
submittal  to  the  RRP. 

It  has  been  the  personal  experience  of  this  author  that  in  addition  to  the  formal  top- 
down  flow  of  urgent  needs,  J/UONs  are  also  created  in  a  grassroots  fashion.  J/UONs  are 
sometimes  the  product  of  warfighters  connecting  directly  with  product  centers  and 
technology  experts.  Once  the  need  is  expressed  and  a  solution  found,  the  J/UON  is  drafted 
and  submitted  through  the  fonnal  channels  and  the  solution  is  presented  to  the  decision 
makers  as  an  option.  While  not  officially  endorsed,  it  does  have  the  benefit  on  reducing  the 
time  in  discovering  a  solution. 
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AFRL  has  recently  formalized  a  process  to  respond  directly  to  urgent  warfighter 


needs.  To  understand  it  in  context  an  overview  of  AFRL  is  warranted.  The  official  mission 


of  AFRL  is  to  discover,  develop  and  deliver  technology  for  insertion  into  the  fighting  force. 
It  accomplishes  this  mission  by  three  Core  Processes.  Core  Process  1  (CPI)  focuses  on 
discovery  and  invests  in  basic  research  that  the  Air  Force  has  determined  will  be  needed  to 
maintain  superiority.  Core  Process  2  (CP2)  matures  and  demonstrates  applied  research 
technologies  that  show  potential  for  insertion  into  the  inventory.  Core  Process  3  (CP3)  has 
been  established  to  meet  urgent  needs  by  providing  direct  communication  between  a  Major 
Command  (MAJCOM)  or  Combatant  Command  (COCOM)  and  the  Lab  to  develop  and 


demonstrate  a  solution  within  one  year. 


Figure  3:  AFRL  Core  Processes  (AFRL  Overview,  Lab  101) 


A  Core  Process  3  project  is  loosely  defined  as  capability  that  is  intended  to  be 


fielded  within  two  years.  The  project  can  either  be  initiated  in  two  ways.  “Technology 


push”  efforts  are  considered  by  the  AFRL  Corporate  Board  and  funded  based  on  the 


perceived  benefits  of  the  proposed  solution.  “Requirements  pull”  projects  allow  a 
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MAJCOM  or  COCOM  to  directly  request  AFRL  funding  of  solutions  to  urgent  needs. 
Figure  4  illustrates  that  process. 
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Figure  4:  CP3  Requirements  Pull  Flowchart  (CP3  Innovation  and  Collaboration,  Lab  101) 


Once  the  user  has  identified  the  need  the  request  is  communicated  by  the 
MAJCOM  leadership  (usually  the  first  general  officer  in  the  user"s  chain).  The  request  is 
sent  to  the  AFRL  commander  for  review  and  tags  the  request  as  a  CP3  project.  A  1-3 
month  study  will  define  the  problem  as  defined  by  the  user,  identify  potential  solutions  and 
defines  the  timeline,  cost  and  manning  required  to  meet  the  request.  The  project  is  then 
reviewed  by  the  AFRL  commander  and  if  approved,  executed  according  to  the  proposal.  A 
demonstration  of  the  capability  is  set  for  7-12  months  from  the  initial  request  and,  if  it 
meets  the  user"s  requirements,  is  fielded.  During  this  year,  transition  leads  and  paths  are 
identified  if  the  project  warrants  inclusion  into  a  program  of  record  (POR). 
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Currently  there  is  no  guidance  on  how  to  manage  a  CP3  program  any  differently 
from  any  other  project  in  the  Lab.  Program  managers  are  usually  chosen  due  to  excelling 
perfonnance  on  previous  projects  and  are  typically  “hard  chargers”  found  in  most 
organizations.  Projects  usually  take  on  the  personality  of  the  program  manager,  or  PM. 
While  a  “get  it  done”  attitude  helps  in  completing  the  paperwork  necessary  to  start  and  run 
a  project,  there  may  be  “blind  spots”  that  PMs  are  missing  that  lead  to  inefficiencies, 
rework  or  not  meeting  schedules  that  are,  by  nature  of  the  organization,  aggressive.  This 
study  seeks  to  discover  any  blind  spots  and  provide  recommendations  to  support  the 
program  manager. 

Prior  Studies  of  Defense  Rapid  Development 

A  recent  Defense  Science  Board  (DSB)  study  on  Fulfilling  Urgent  Operational 
Needs  cited  eight  studies  in  the  past  five  years  that  propose  changes  that  would  create  an 
agile,  responsive  process  for  the  DoD  to  rapidly  field  solutions  to  urgent  needs.  The 
common  theme  was  that  the  current  acquisition  model  does  not  satisfy  the  timeline  of 
developing  solutions  to  urgent  needs  because  it  focuses  on  “micromanaging  risk  and 
achieving  the  100  percent  solution”  (DSB,  2009).  While  it  might  be  tempting  to  use  this  as 
justification  to  abandon  systems  engineering  principles  to  reduce  timelines  and  “get  it 
faster”,  this  impulse  should  be  resisted.  The  fact  is  technical  solutions  are  required  to  be 
engineered  to  fit  within  the  larger  military  toolkit.  Solutions  must  not  work  only  in  a 
vacuum. 

The  DSB  also  published  a  study  establishing  the  Strategic  Technology  Vectors 
(DSB,  2007).  A  recommendation  is  made  for  a  single  Rapid  Fielding  Office  to  coordinate 
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all  rapid  reaction  organizations.  This  call  is  later  echoed  by  the  DSB  Task  Force  on 
Fulfilling  Urgent  Needs  with  the  proposal  of  the  Rapid  Acquisition  Fielding  Agency  (DSB, 
2009).  One  key  recommendation  in  both  reports  is  the  use  of  systems  engineering  as  a 
“cross-cutting  enabler”  that  “manages  the  tradeoffs  necessary  to  develop  and  field  a  system 
that  is  affordable,  is  sustainable,  is  delivered  on  schedule,  satisfies  user  needs,  and 
minimizes  risk.”  (DSB,  2007) 

The  search,  then,  is  to  find  the  proper  balance  of  SE  practices  implemented  in  a 
rapid  reaction  project.  Two  recent  studies  have  independently  addressed  a  tailored  SE 
approach  for  AFRL  and  rapid  development  and  acquisition.  The  2009  AFIT  thesis,  “A 
Tailored  Systems  Engineering  Framework  for  Science  and  Technology  Projects,” 
addressed  a  perceived  disconnect  between  the  conceptual  framework  of  SE  and  its  actual 
implementation  in  research  projects.  It  produced  a  SE  tool  based  on  six  discriminates: 
budget  category,  budget  size,  core  process,  technology  readiness  level,  level  of  integration, 
and  requirements  maturity  (i.e.  requirements  push  or  tech  pull).  The  output  of  the  tool  when 
applied  to  a  particular  project  was  a  level  of  “SE  rigor”  which  placed  each  technical  and 
technical  management  process  into  one  of  the  following  categories:  Required, 
Recommended,  Watch  List,  Not  Applicable.  It  follows  that  this  tool  could  be  applied  to 
rapid  reaction  projects. 

In  the  2008  AFIT  thesis,  “An  Analysis  of  Methodologies  and  Best  Practices  for 
Rapidly  Acquiring  Technologies  to  Meet  Urgent  Warfighter  Needs”,  Capt  David  Solomon 
studied  the  ways  in  which  rapid  reaction  organizations  foster  innovation  and  how  they 
utilize  the  “critical  enablers”  identified  in  the  DSB  Strategic  Technology  Vector  study.  As 
one  of  the  enablers,  systems  engineering"s  perceived  value  varied  across  DoD  rapid 
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reaction  organizations.  Some  respondents  viewed  it  as  vital  while  others  saw  no  value. 
Nevertheless,  one  recommendation  for  future  study  was  to  investigate  the  appropriate 
amount  of  systems  engineering  and  detennine  a  tailored  approach  for  projects  that  answer 
an  urgent  need. 

Rapid  Development  Outside  the  Defense  Department 

Rapid  development  outside  the  military  encompasses  a  few  different  areas.  The 
field  of  prototyping  deals  with  manufacturing  a  test  product  quickly.  While  the  goal  of  this 
study  is  to  shorten  the  time  to  deliver  products,  rapid  prototyping  focuses  on  the  specific 
assembly  of  the  product  based  on  a  complete  design.  Therefore  the  scope  of  this  inquiry 
will  extend  beyond  rapid  prototyping.  The  field  of  software  development  also  offers  many 
methods  to  deliver  products  quickly.  The  attributes  of  software  development  lend  to  short 
time  cycles  from  product  design  and  integration  to  test  and  typically  go  through  numerous 
iterations  before  a  final  product  is  delivered.  While  many  of  the  urgent  needs  of  the  military 
tend  towards  hardware  solutions,  the  lesson  of  understanding  a  problem  and  developing  a 
solution  is  common  to  both  cases.  Finally,  there  exist  studies  in  business  management 
literature  on  how  to  shorten  the  cycle  of  product  development.  While  the  focus  of  these 
studies  is  to  be  first  to  market  and  being  responsive  to  shifting  consumer  trends  in  order  to 
maximize  profitability,  the  methods  uncovered  will  also  be  applicable  since  the  goal  of  a 
short  product  cycle  is  common  to  both.  Also,  being  first  to  market  in  the  military  sense 
equates  to  an  advantage  in  technology  or  tactics  for  which  the  enemy  hasn't  developed 
countenneasures. 

In  1991,  James  Martin  wrote  Rapid  Application  Development  which  sought  to 
provide  an  alternative  to  “rigid”  development  methods,  such  as  the  waterfall  method,  to  the 
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software  development  community  (Howard,  2002).  In  1995,  a  group  of  software 
developers  in  the  UK  teamed  with  end-user  organizations  to  create  the  Rapid  Application 
Development  (RAD)  standard  (Millington  and  Stapleton,  1995).  The  goal  of  the  standard 
was  to  be  a  framework  for  vendors  to  follow  when  writing  software  applications.  They 
listed  live  phases  in  the  development  life  cycle.  The  first  two  focused  on  business  matters, 
namely  a  feasibility  report  and  a  business  study.  The  third  phase  focused  on  “functional 
model  iteration,  producing  a  functional  prototype,  a  statement  of  non-functional 
requirements  and  an  implementation  strategy”.  Following  this  was  a  “design-and-build 
iteration”  where  the  prototype  was  tested  against  requirements.  The  authors  suggest  three 
iterations  between  the  third  and  forth  phases  “for  initial  investigation,  refinement,  and 
consolidation.”  Finally  the  system  is  implemented  with  the  users  with  manuals  and 
training. 

While  touted  as  a  standard,  RAD  was  more  of  a  philosophy  requiring  autonomy 
and  senior  leader  buy-in,  experience  and  recognized  talent  among  a  stable  team.  In  2000,  a 
case  study  was  used  to  showcase  RAD  methods  in  an  internal  BT  (fonnerly  British 
Telecom)  intranet  project  (Beynon-Davies,  et  al,  2000).  In  it,  a  matrixed  team  of  employees 
was  separated  from  their  duty  stations  and  tasked  with  building  an  internal  on-line  resource 
for  their  corporate  relations  department.  In  it  the  team  delivers  a  working  prototype  in  three 
weeks.  While  not  being  the  final  answer,  the  product  allowed  for  future  enhancement  based 
on  the  potential  of  added  requirements  and  was  a  complete  product  in  that  it  met  the 
requirements  within  the  scope  of  the  project. 

A  group  at  BAE  Systems,  Advanced  Technology  Centre,  took  RAD  one  step 
further  in  aggregating  Extreme  Programming  (one  of  the  different  flavors  of  agile  software 
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development  including  RAD)  and  systems  engineering  (Jones  and  Leung,  2005).  The 
objective  was  to  build  a  wide  area  surveillance  system  that  could  be  commercialized 
despite  technical  complexity.  Their  approach  to  the  problem  followed  a  three-point  plan:  1) 
Define  the  CONOP  to  give  a  high  level  view  of  the  system  and  provide  scenarios  for 
development  and  test.  2)  Detennine  key  questions  in  understanding  the  requirements.  And 
3)  Develop  a  prototype  system  based  on  technologically  mature  components.  The  case 
study  documents  the  development  of  a  prototype  system  that  through  iterations  from  an 
initial  system,  meets  the  perfonnance  parameters  established  at  the  beginning  of  the 
project.  Key,  in  the  authors”  minds,  was  the  understanding  of  the  component  technology 
and  the  requirements  of  the  system,  both  individual  perfonnance  requirements  and  an 
understanding  of  the  scenarios  in  which  it  was  to  perfonn. 

In  the  world  of  business  literature,  a  series  of  books  written  in  the  1990s  set  the 
stage  for  companies  to  think  about  how  they  develop  new  products.  In  1992,  Wheelwright 
and  Clark  present  “concepts  for  the  effective  organization  and  management  of  product  and 
process  development”  in  their  book,  Revolutionizing  Product  Development.  Using  case 
studies  of  Kodak,  GE,  Motorola  and  Lockheed  they  looked  at  project  management 
frameworks  in  each  company  and  identify  five  commonalities,  namely  “customer  focus, 
discipline,  coherence,  fit  and  sharing  the  pattern.”  Customer  focus  sought  to  understand  the 
user's  requirements  but  also  their  unmet  needs.  A  contemporary  case  in  the  consumer 
electronics  market  would  be  Apple"s  success  with  the  iPod.  In  the  military  sense,  this  can 
be  seen  as  understanding  the  capability  gap  beyond  what  the  user  states  as  requirements. 
“What  aspect  of  their  mission  is  not  being  met  that  they  don't  know  yet?”  Discipline  is 
geared  towards  a  streamlined  process  that  fosters  “thoroughness  and  consistency”  but 
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doesn't  “stifle  creativity”  or  slow  down  projects  with  unnecessary  oversight.  Coherence 
deals  with  assigning  the  right  skills  to  projects  not  only  in  a  technical  sense  but  also  a 
managerial  approach.  Fit  with  the  mission  ensures  that  the  solutions  of  a  project  match  the 
stated  objectives;  for  example,  technical  solutions  to  solve  technical  problems  rather  than 
manufacturing  or  personnel  issues.  Finally,  sharing  the  pattern  looks  at  how  organizations 
communicate  a  common  framework  and  set  expectations  of  “what  must  be  done,  when  and 
how.” 

Focusing  on  the  “discipline”  aspect,  Wheelwright  and  Clark  devote  Chapter  9  to 
“Tools  and  Methods”  for  executing  projects.  After  proposing  strategies  for  meeting 
perfonnance  goals  and  laying  out  effective  plans,  making  sure  the  right  people  understand 
the  right  processes,  the  authors  focus  in  on  problem  solving  at  the  working  level.  Cross¬ 
functional  teams  meeting  with  each  other  to  “solve  specific  problems.”  In  their  study,  the 
ability  to  solve  problems  was  at  the  heart  of  good  product  development.  Their  method  for 
consistent,  quality  problem  solving  is  broken  down  into  the  “design-build-test  cycle”. 
Similar  to  the  Vee  Model  in  the  Systems  Engineering  Handbook  (INCOSE,  2003), 
Wheelwright  and  Clarkes  model  is  straightforward.  In  the  design  phase,  requirements  and 
tradeoffs  are  explored  with  clear  objectives  and  alternative  solutions  are  generated.  The 
build  phase  then  acts  on  those  designs  whether  producing  CAD  models  or  test  code  or  other 
engineering  prototypes.  The  test  phase  then  executes  a  test  plan  based  on  collecting  the 
right  infonnation  accurately  in  an  environment  as  close  as  possible  to  the  intended  use 
enviromnent.  Wheelwright  and  Clark  also  encourage  the  use  of  iterations  if  the  first  cycle 
fails  to  meet  the  expected  perfonnance  measures. 
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In  1993,  in  the  second  edition  of  his  book,  Winning  at  New  Products:  Accelerating 
the  Process  from  Idea  to  Launch,  Robert  Cooper  proposed  Stage-Gates  as  a  method  to 
develop  new  products.  Similar  to  the  Defense  acquisition  milestones,  Stage-Gates  take  an 
idea  through  a  series  of  gates  to  detennine  if  the  idea  is  worth  committing  resources  to  go 
to  the  next  stage.  The  five  stages,  being  Preliminary  Design,  Business  Case,  Development, 
Testing  and  Validation,  Full  Product  &  Market  Launch,  all  have  entrance  and  exit  criteria 
and  allow  a  team  to  focus  on  each  phase  before  proceeding  to  the  next.  One  caution  from 
the  author  is  that  it  is  intended  that  the  process  not  focus  on  one  functional  area  per  stage, 
but  rather  use  multi-functional  teams  to  work  in  parallel  through  the  stages.  For  instance,  a 
test  team  member  might  have  valuable  insight  on  creating  a  testable  specification  during 
the  preliminary  design  phase  that  will  shorten  test  time  down  the  road  or  a  marketing  team 
member  may  use  validation  results  that  would  target  key  early  adopters  in  winning  early 
critiques  of  the  product.  Stage-Gates  are  not  intended  to  be  inflexible  and  companies  are 
encouraged  to  tailor  them  to  suit  different  project  category  needs.  The  process  should 
enable  teams  to  create  the  right  product  efficiently,  not  enable  management  to  become 
roadblocks. 

For  the  purposes  of  this  research,  the  reader  should  assume  that  the  project  idea  has 
been  identified,  the  business  case  has  been  pitched  and  management  has  agreed  to  initiate 
the  project.  This  starts  the  project  in  Stage  3,  Development.  The  first  action  Cooper 
suggests  is  to  confirm  the  requirements.  Bring  users  in  to  expose  any  incorrect  assumptions 
or  to  re-prioritize  the  perfonnance  targets  in  case  there  were  shifts  in  the  marketplace 
(analogous  to  threats  and  capability  gaps  in  the  military).  A  development  plan  is  then  built 
with  tasks  listed,  realistic  timelines  to  complete  and  resources  assigned.  Additionally, 
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milestones  throughout  the  project  with  definable  goals  for  the  development  set  targets  that 
team  members  can  agree  on  and  plan  activities  accordingly.  Throughout  the  development 
stage,  in-house  testing  is  iterative  and  customer  involvement  and  feedback  is  essential  in 
creating  the  right  product.  Cooper"s  Stage  4  encompasses  preference  tests  (“do  I  like  your 
product  better  than  what  I  have”)  and  field  trials  (limited  quantities  or  simulations  to  gauge 
user  interest)  are  analogous  to  the  DoD"s  operational  testing.  In  each  case,  the  product  is 
tested  in  a  relevant  enviromnent  with  representative  users  to  validate  the  product  and 
collect  feedback  before  full-rate  production. 

Five  years  later  in  1998,  the  second  edition  of  Preston  Smith  and  Donald 
Reinersten"s  book,  Developing  Products  in  Half  the  Time:  New  Rules,  New  Tools  was 
published.  Smith  and  Reinertsen  pick  up  on  the  ideas  of  the  two  prior  authors  and  offer  a 
few  of  new  concepts  such  as  the  “fuzzy  front  end”  which  they  define  as  the  time  from  when 
an  opportunity  is  discovered  to  when  a  development  team  begins  work  on  the  project. 

Some  of  this  time  is  due  to  the  bureaucratic  nature  of  organizations  but  also  includes 
factors  such  as  expedited  shipping  costs  that  may  be  reduced  if  identified  earlier  in  the 
project  timeline.  As  an  improvement  to  Cooper"s  Stage-Gate  style  of  phased  project 
planning,  Smith  and  Reinertsen  suggest  product  development  organizations  reduce  the 
emphasis  on  the  gates  (i.e.  less  fonnal  reviews)  and  more  emphasis  on  the  flow  of  the 
project.  For  example,  management  may  decide  exit  criteria  for  each  stage  and  let  the  team 
decide  when  they  have  met  those  criteria.  Reports  could  be  presented  at  quarterly  reviews 
(or  more  frequent)  to  ensure  projects  don't  run  amok. 

In  separate  sections,  Smith  and  Reinertsen  address  managing  risk.  In  the  first, 
technical  risk  is  addressed  by  focusing  risk  mitigation  within  individual  models  of  a  design 
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versus  distributing  risk  management  broadly  over  the  whole.  System  integration  risk  is 
reduced  by  focusing  on  the  riskiest  modules  and  the  use  of  margins  or  safety  factors  for 
critical  items  such  as  fastener  tolerances.  In  the  second  section  (Chapter  12),  the  authors  re¬ 
address  technical  risk  and  compare  its  relation  to  market  risk.  Too  much  technical  risk 
management,  in  the  fonn  of  multiple  reviews,  can  increase  market  risk  by  delaying  a 
project  introducing  uncertainty  into  sales  predictions.  Of  course,  not  enough  technical  risk 
management  results  in  costly  surprises  late  in  the  development  cycle.  Smith  and  Reinertsen 
suggest  that  it  depends  on  the  project  goals  and  a  moderate  level  of  risk  management  leads 
to  shorter  development  cycle  times.  To  control  risk,  the  authors  offer  a  commonly  used 
chart  where  the  probability  of  an  event  (i.e.  assembly  of  the  engineering  prototypes)  is 
compared  with  the  impact  of  it  not  happening  (i.e.  field  tests  slip  1  month)  as  shown  in 
Table  2.  Events  that  are  likely  to  occur  with  a  high  impact  on  schedule,  say,  are  identified 
as  high  risk. 


Table  2:  Risk  Analysis  Table 
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In  2007,  Michael  Grieves  wrote  Product  Lifecycle  Development  based  on  his 
background  in  the  automotive  and  IT  industries.  While  not  specifically  focused  on  “rapid 
development”,  Grieves  notes  that  cycle  times  are  an  external  driver  pushed  by  customers 
and  competition  and  provides  examples  of  multiple  industries  that  exhibit  this  phenomenon 
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from  the  automobile  to  fashion  to  phannaceuticals.  His  describes  the  five  functional  areas 
of  product  lifecycles  as  “plan,  design,  build,  support,  dispose.”  The  planning  consists  of 
“requirements  analysis  and  planning”  which  leads  into  the  design  phase  where  engineers 
play  with  trade-offs  while  ensuring  the  requirements  are  met.  Prototypes  are  then  built 
based  on  specifications  that  ensure  the  “various  components  fit  together  in  an  integrated 
system  and  that  the  system  is  internally  consistent.”  During  the  build  phase,  manufacturing 
engineers  decide  how  the  product  is  built  and  in  what  steps.  Finally  the  support  and 
disposal  ensure  feedback  is  attained  from  the  customer  and  decisions  are  made  concerning 
what  to  do  with  it  after  its  use. 

Finally,  the  Product  Development  and  Management  Association  (PDMA)  has 
published  a  collection  of  tools  “most  appropriate  for  use  in  the  engineering  design  and 
development  phases”  of  new  product  development  in  The  PDMA  Toolbook  3  for  New 
Product  Development.  Released  in  2007,  PDMA  Toolbook  covers  multiple  topics  from 
trade-off  analysis  to  intellectual  property  to  development.  Gregory  Githens  offers  the 
Rolling  Wave  approach  to  development  cycles.  In  traditional  projects,  a  schedule  is  built 
and  tasks  are  populated  from  beginning  to  end  on  the  assumption  that  all  actions  required 
are  “knowable”  upfront.  The  Rolling  Wave  approach  seeks  to  create  a  “robust"  schedule 
that  is  flexible  and  overcomes  “brittle  schedules”  that  occur  when  early  slips  lead  to 
“individuals  [that]  narrow  their  focus  to  their  own  subjective  view  of  priorities”. 

The  proposed  solution  seeks  a  “plan-do-plan-do”  series  of  activities  where 
segments  of  the  project  are  broken  up  into  “rough  order  magnitudes”  or  “ROMs”.  Tasks 
are  only  planned  out  as  the  team  reaches  each  ROM  Group.  The  argument  goes  that  task 
completion  dates  grow  in  uncertainty  the  farther  away  they  are.  Therefore,  focusing  on 
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short-term  forecasts  will  produce  more  reliable  estimates  in  the  long  run.  Githens 
emphasizes  that  agility  both  in  the  product  and  project  architecture.  Product  agility  covers 
interface  management  and  the  use  of  common  technical  standards.  Project  agility  concerns 
team  composition,  levels  of  authority,  review  and  approval  cycles,  roles  and 
responsibilities,  risk  and  issues  analysis,  escalation  strategy,  etc.”  Githens  then  offers  six 
steps  in  executing  a  Rolling  Wave  approach  as  shown  in  Figure  5. 


Team 

Charter 


\F 


Prioritize 
on  Risk 


Figure  5:  Rolling  Wave  Methodology 


Step  1  starts  with  a  team  charter  that  captures  the  requirements  management,  team 


roles  and  responsibilities  and  lays  out  the  vision  of  the  project.  Step  2  is  creating  a  work 


breakdown  structure  (WBS)  to  include  Level  1  and  Level  2  activities.  Level  1  will  be 


segmented  by  the  ROM  Groups  or  “planning  windows”.  Tasks  that  are  well  understood 
and  can  be  detailed  to  the  work  package  level  are  documented  in  the  WBS  while  less 
certain  tasks,  say  specifications  that  require  user  feedback  on  prototypes,  are  left  undefined 
and  assigned  to  later  planning  windows.  Step  3  details  the  tasks  of  the  WBS  for  each  and 
starts  the  “plan  a  little”  stage.  Cost  and  schedules  are  presented  as  range  estimates  that  can 
be  as  little  as  half  the  target  or  as  much  as  twice  the  target.  This  requires  project  managers 
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to  gain  the  trust  of  stakeholders  to  allow  the  flexibility  of  this  planning  approach. 
Additionally,  a  transition  plan  should  be  fonnulated  to  prevent  delays  at  the  end  of 
development  and  the  beginning  of  product  launch  or  delivery.  Step  4  proposes  managing 
risk  and  reprioritizing  activities  versus  establishing  a  baseline  and  monitoring  variance. 

Step  5  executes  the  first  ROM  Group  or  “do  a  little”  with  attention  paid  to  monitoring  risks 
and  keeping  key  stakeholders  up  to  date  if  cost  or  schedule  variances  break  a 
predetennined  threshold.  Step  6  iterates  the  planning-doing  cycles  until  completion.  The 
key  activities  are  assessing  the  groups"  progress,  anticipating  tasks  in  future  planning 
windows  and  ensuring  the  “big  picture”  is  fixed  in  each  team  members"  decision  making 
processes. 

Summary 

This  chapter  presented  an  overview  of  rapid  development  efforts  within  the  DoD, 
the  USAF  and  AFRL.  In  addition  it  looked  to  industry  for  additional  frameworks  for  rapid 
development.  These  and  other  methods  from  literature  will  be  compiled  and  consolidated  in 
Chapter  3.  The  common  processes  will  be  translated  into  the  SE  technical  and  technical 
management  processes  to  take  advantage  of  acquisition  training  required  for  certification  in 
the  acquisition  career  field.  A  purposive  sampling  of  AFRL  rapid  development  team 
members  will  identify  the  methods  currently  used  in  the  CP3  organization. 
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III.  METHODOLOGY 


Introduction 

A  mixture  of  qualitative  and  quantitative  methods  will  be  used  to  assess  the  relative 
importance  of  various  processes  and  artifacts  defined  in  Chapter  4,  Systems  Engineering, 
of  the  Defense  Acquisition  Guidebook  (DAG)  with  respect  to  rapid  development  projects. 
Terms  referenced  in  the  definitions  of  each  process  will  be  compared  with  those  found  in 
the  literature  and  evaluated  based  on  importance.  A  management  strategy  will  then  be 
proposed  based  on  the  relative  importance  of  each  process.  A  comparison  with  key 
activities  identified  by  a  purposive  sampling  of  AFRL  rapid  development  team  members 
will  provide  an  evaluation  of  the  proposed  strategy  compared  to  recent  projects. 

With  origins  in  the  product  development  community,  methodologies  such  as  Stage 
Gating,  Product  Lifecycle  Management  and  Rapid  Application  Development  offer 
strategies  to  apply  to  rapid  development.  Identifying  key  processes  in  these  methods  will  be 
compared  to  the  8  technical  and  8  technical  management  processes  as  defined  in  the  DAG. 
Product  Development  Literature  Evaluation 

Content  analysis  is  a  method  that  has  origins  in  the  1940s  and  began  with 
conducting  word  counts  on  texts.  Eventually  it  matured  to  concepts  and  meanings.  A 
conceptual  analysis  is  a  sub-category  of  content  analysis  where  texts  are  examined  for 
frequency  of  words  or  phrases  related  to  a  research  question.  In  this  study,  product 
development  texts  will  be  examined  for  the  frequency  of  SE  keywords  determined  by  this 
author  from  the  DAG  SE  process  definitions.  The  relative  frequency  of  these  keywords  will 
provide  insight  as  to  what  the  authors  deem  important  in  instructing  the  reader  how  to  do 
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product  development.  A  listing  of  keywords  for  the  technical  management  and  technical 
processes  can  be  found  in  Tables  4  and  5,  respectively. 

Processes  will  be  identified  and  assessed  in  each  literature  method  based  on  their 
importance.  Importance  in  this  case  will  be  equated  with  frequency.  The  number  of 
references  to  keywords  from  the  DAG  SE  process  definitions  will  be  nonnalized  and 
assigned  an  importance  score  ranging  from  1-  Not  Important  to  5-  Extremely  Important  as 
seen  in  Table  5.  Keywords  are  listed  in  Tables  4  and  5,  respectively.  The  reader  is  referred 
to  the  DAG,  Chapter  4,  section  4.2.3,  for  complete  definitions  of  each  technical  and 
technical  management  process. 

Table  3:  Keywords  of  the  Technical  Management  Processes 


Technical  Management  Process 

Keywords 

Decision  Analysis 
(DA) 

Trade  Studies,  Analyses-  Alternatives,  Supportability, 
Cost,  Trade  Off 

Technical  Planning 
(TP) 

Scope  of  Technical  Effort,  Systems  Engineering  Plan 

Technical  Assessment 

Technical  Review,  Program  Review,  Technical 

(TA) 

Interchange,  Interface  Control  Working  Group 

Requirements  Management 
(Req  Mgmt) 

Requirements,  Traceability,  Change  Management 

Risk  Management 

Risk-  Identification,  Analysis,  Mitigation,  Tracking 

Configuration  Management 
(Config  Mgmt) 

Technical  Baseline,  Functional  Baseline,  Allocated 
Baseline,  Product  Baseline,  Change  Management, 

Audits 

Data  Management 

Technical  Data,  Records,  Organization,  Sharing 

Interface  Management 
(Int  Mgmt) 

Interface  Specifications,  Standards,  Compliance 
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Table  4:  Keywords  of  the  Technical  Processes 


Technical  Process 

Keywords 

Stakeholders  Requirements 
Definition  (SRD) 

Requirements,  CONOPS,  Constraints,  Stakeholder 

Requirements  Analysis 
(RA) 

Functional  Analysis,  Performance  Requirements, 

Functional  Architecture 

Architecture  Design 
(AD) 

Design  Solutions,  Logical  Models  or  Views,  Physical 
Architecture,  Specification 

Implementation  (Imp) 

System  Elements,  Production,  Component  Testing 

Integration  (Int) 

Assembly,  Interfaces,  Incorporation,  Prototype 

Verification  (Ver) 

Demonstration,  Inspection,  Analysis,  Test 

Validation  (Val) 

Validation,  Evaluation 

Transition  (Trans) 

Installation,  Integration,  Fielding 

Table  5:  Importance  Scale  Descriptions 


Importance  Scale 

Description 

0-1 

Not  Important 

1-2 

Somewhat  Important 

2-3 

Important 

3-4 

Very  Important 

4-5 

Extremely  Important 

Proposed  Framework 

A  comparison  of  each  method  against  the  16  system  engineering  processes 
endorsed  by  Chapter  4  of  the  DAG  will  highlight  where  emphasis  has  been  given  by  the 
authors.  Keywords  will  be  counted  for  frequency  within  the  books  and  texts  to  indicate 
importance  and  normalized  to  provide  comparable  values.  Normalized  scores  are  calculated 
by  the  equation: 
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Where  Scores  is  the  number  of  keyword  references  from  each  author  for  a  process,  i, 
MAX(Scorej)  is  the  maximum  number  of  references  from  each  author  in  a  particular 
process  and  multiplied  by  5  to  fit  the  scale  in  Table  5.  This  will  nonnalize  the  scores  to 
show  which  is  process  is  most  emphasized  which  will  be  equated  with  most  important.  The 
scores  for  the  technical  management  and  the  technical  processes  will  be  nonnalized 
separately  since  technical  processes  typically  take  place  sequentially  and  technical 
management  processes  occur  throughout  the  life  of  a  project.  Scores  will  then  be  totaled 
and  calculated  as  a  percentage  to  display  relative  weights  to  which  program  managers  can 
allocate  resources  (time,  money,  and  people). 

Purposive  Sampling  and  Analysis 

This  study  will  follow  a  similar  methodology  conducted  by  a  recent  INCOSE  paper 
(Mulheam  and  Brouse,  2011).  In  it,  the  authors  investigate  small  infonnation  technology 
(IT)  projects  with  the  intent  of  filtering  the  most  important  documents  to  “effectively  and 
efficiently  manage  the  project”.  Twelve  knowledge  areas  were  combined  from  ontologies 
from  both  the  program  management  and  systems  engineering  literature  to  encompass  the 
technical  emphasis  of  the  IT  projects.  “Small”  IT  projects  were  defined  as  “under  12 
months  in  duration  and  cost  less  than  $  1 ,5M.”  A  survey  sent  to  a  purposive  sampling  of  IT 
professionals  identified  the  top  15%  of  documents  and  reviews.  It  is  the  intent  of  this 
review  to  conduct  the  same  evaluation  with  AFRL  program  managers  for  rapid  reaction 
projects. 
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Interviews  with  key  personnel  (program  managers  and  chief  engineers)  on  rapid 
development  projects  will  provide  an  evaluation  of  the  current  emphasis  placed  on  each 
process.  Each  process  will  be  assessed  by  the  author  on  a  1-5  scale  of  importance.  The 
criteria  are  derived  from  SE  technical  process  outputs  as  outlined  in  the  INCOSE  Systems 
Engineering  Handbook,  v3.1.  These  definitions  were  chosen  over  standard  DoD 
Acquisition  tenninology  to  encompass  activities  that  met  the  intent  but  weren't  specifically 
defined  by  DoD  terms.  Some  criteria  were  augmented  by  DoD  Developmental  and 
Operational  Test  activities  where  the  INCOSE  SE  handbook  provided  insufficient 
measures  to  stratify  the  fonnality  of  a  particular  process  (i.e.  Verification  and  Validation). 
A  listing  of  interview  questions  and  topics  is  found  in  Appendix  A.  These  scores  will  then 
be  compared  with  the  model  detennined  by  the  product  development  literature. 
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IV.  DATA  ANALYSIS 


Overview 

This  chapter  will  present  the  results  of  the  conceptual  analysis  of  the  product 
development  literature  with  respect  to  DAG  SE  processes.  From  this  analysis,  an  allocation 
of  resources  will  be  presented  by  looking  at  the  relative  scores  of  each  process.  The  chapter 
will  then  present  the  results  of  the  levels  of  fonnality  of  SE  processes  uncovered  in  the  case 
studies.  Finally,  the  two  levels  of  importance  will  be  compared  with  each  other  and 
analyzed. 

Results 

Tables  1 1  and  12  show  the  nonnalized  scores  of  the  SE  processes  found  in  each 
rapid  development  approach.  Figures  6  and  7  display  this  data  as  bar  charts,  with  the 
standard  deviation  computed  for  the  error  bars.  Based  on  the  data,  the  most  important 
Technical  Management  processes  for  rapid  development  are  Technical  Planning,  Decision 
Analysis,  Risk  Management  and  Technical  Assessment.  There  is  a  general  concurrence  that 
Technical  Planning  is  a  must  for  product  development  as  this  process  has  the  highest  score 
with  one  of  the  smallest  deviations.  All  other  Technical  Management  data  show  a  mixed 
emphasis  for  each  of  the  other  processes.  Decision  Analysis,  Technical  Assessment,  Risk 
Management  are  slightly  more  emphasized  while  Requirements  Management, 
Configuration  Management  and  Data  Management  slightly  less  and  Interface  Management 
almost  not  at  all. 

The  most  important  Technical  Processes  are  Stakeholder  Requirements  Definition, 
Architecture  Design  and  Integration.  There  is  a  concurrence  that  Stakeholder  Requirements 


30 


Definition  is  emphasized  heavily  while  Architecture  Design,  Integration  and  Verification 
are  emphasized  slightly  more  and  Validation  slightly  less.  Requirements  Analysis, 
Implementation  and  Transition  received  low  scores  having  not  been  emphasized  in  the 
texts. 


Table  6:  SE  Technical  Management  Process  Scores 


Technical 

Management 

Processes 

DA 

TP 

TA 

Req 

Mgt 

Risk 

Mgt 

Config 

Mgt 

Data 

Mgt 

I/F 

Mgt 

Wheelright 
and  Clark 

2.50 

5.00 

1.50 

0.50 

1.00 

0.00 

0.00 

0.00 

Cooper/  Stage 
Gates 

4.17 

5.00 

3.33 

0.00 

3.33 

0.00 

0.00 

0.00 

Smith  and 
Reinertsen 

2.14 

5.00 

1.43 

1.43 

5.00 

1.43 

1.43 

0.71 

PLD 

1.25 

3.75 

0.00 

1.25 

0.00 

3.75 

5.00 

0.00 

RAD 

1.67 

5.00 

1.67 

3.33 

1.67 

3.33 

3.33 

0.00 

PDMA 

4.38 

4.38 

5.00 

0.63 

3.13 

0.63 

0.63 

0.63 

AVERAGE 

2.68 

4.69 

2.15 

1.19 

2.35 

1.52 

1.73 

0.22 

St  Dev 

1.30 

0.52 

1.75 

1.17 

1.81 

1.65 

2.03 

0.35 

Table  7:  SE  Technical  Process  Scores 


Technical 

Processes 

SRD 

RA 

AD 

Imp 

Int 

Ver 

Val 

Tran 

s 

Wheelright 
and  Clark 

5.00 

1.67 

3.33 

1.67 

2.92 

2.50 

2.50 

0.00 

Cooper/  Stage 
Gates 

5.00 

0.56 

1.67 

0.56 

1.11 

2.22 

1.67 

1.11 

Smith  and 
Reinertsen 

2.22 

1.11 

5.00 

0.00 

1.67 

0.56 

0.00 

0.56 

PLD 

3.75 

1.25 

3.75 

0.00 

5.00 

2.50 

1.25 

0.00 

RAD 

5.00 

1.67 

1.67 

0.00 

1.67 

3.33 

1.67 

0.83 

PDMA 

5.00 

0.34 

0.86 

0.17 

0.34 

0.34 

0.34 

0.52 

Average 

4.33 

1.10 

2.71 

0.40 

2.12 

1.91 

1.24 

0.50 

St  Dev 

1.15 

0.55 

1.57 

0.66 

1.64 

1.19 

0.93 

0.44 
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Figure  6:  Technical  Management  Process  Scores  with  Standard  Deviation  Error 
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Figure  7:  Technical  Process  Scores  with  Standard  Deviation  Error 
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Analysis 

From  the  emphasis  the  literature  places  on  the  technical  management  processes,  the 
project  manager  must  first  manage  the  scope  and  develop  the  technical  plan  for  achieving 
the  project  objectives.  This  is  a  logical  conclusion  since  vague  requirements  or  a  lack  of 
technical  direction  can  lead  to  miscommunication  or  mismanaged  expectations  which  can 
lead  to  rework  or  product  rejection.  This  does  not  mean  that  the  initial  scope  or  technical 
plan  must  change,  rather  that  the  project  manager  must  manage  how  potential  changes 
affect  the  development  of  the  end  product.  Once  the  technical  planning  process  is  in  place, 
the  project  manager  must  support  it  with  Decision  Analysis,  Technical  Assessments  and 
Risk  Management.  In  other  words,  to  direct  the  technical  effort  a  project  manager  must 
understand  the  perfonnance  and  cost  trade-offs  of  different  approaches,  assess  the  progress 
throughout  the  development  and  have  a  robust  risk  management  procedure-  identify, 
assess,  mitigate,  track-  to  deal  with  problems  before  they  come  to  bear.  Requirements 
Management,  Configuration  Management,  Data  Management  and  Interface  Management 
are  all  things  a  project  manager  should  be  mindful  of,  however,  they  should  not  take  a 
majority  of  his/her  time  and  resources. 

The  Technical  Management  scores  generally  follow  the  “Design-  Build-  Test 
Cycle”  put  forward  by  Wheelright  and  Clark.  Stakeholder  Requirements  Definition  was 
clearly  the  most  important  which  is  logical  since  new  products  are  developed  to  meet  a 
need,  whether  a  perceived  need  in  the  commercial  industry  or  a  stated  need  in  the  defense 
acquisition  system.  A  project  manager  must  understand  how  the  new  product  is  intended  to 
perfonn,  in  what  environment,  how  it  interacts  with  other  systems  and  so  on.  Architecture 
Design  followed  next  with  a  physical  solution  to  meet  the  requirements.  Having  CAD 
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models  and  logical  views,  such  as  the  DoDAF  Architecture,  ensures  that  the  team  is  “on  the 
same  page”  for  designing  the  system.  Integration  and  Verification  were  next  in  importance 
showing  that  it  is  important  to  put  the  components  together  correctly  and  test  the  system  to 
ensure  it  was  built  right. 

The  lower  Technical  Management  process  scores  could  be  explained  as  an  outcome 
of  focusing  on  the  higher  scored  processes.  For  example,  if  a  project  clearly  defines  the 
outputs  of  the  Stakeholder  Requirements  Definition  process-  namely  the  CONOPS, 
enviromnent,  constraints  as  stated  by  all  the  stakeholders-  the  Validation  testing  should  be 
easier  and  thus  less  emphasized.  If  the  Architecture  Design  is  correct,  then  the 
Implementation  of  building  the  components  to  the  design  specifications  should  be  well 
understood  and  less  emphasized.  If  the  right  product  is  built  correctly,  then  Transition 
should  follow  without  major  problems.  The  score  for  Requirements  Analysis  however 
seems  out  of  place.  This  result  could  be  explained  by  its  dependence  on  a  successful 
Stakeholder  Requirements  Definition  phase,  but  it  is  also  conceivable  that  the  keywords 
chosen  were  an  inaccurate  measure  of  the  process  or  that  the  keywords  exist  primarily 
within  the  systems  engineering  community  or  specific  to  DoD  SE  that  the  authors  of  the 
texts  under  study  did  not  use  this  tenninology. 

By  converting  the  scores  to  an  overall  percentage,  as  shown  in  Figures  8  and  9,  a 
program  manager  can  weigh  each  process  relative  to  the  other  and  plan  out  a  project.  Since 
these  are  process  resource  allocations  it  makes  more  sense  to  apply  these  percentages  to  the 
management  of  a  project  and  not  the  overall  budget,  which  could  include  high-cost  items.  It 
can  be  helpful  to  think  of  applying  the  percentages  to  the  time  allotted  during  regular 
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meetings  or  hours  in  a  week  that  a  project  manager  focuses  his/her  time  directing  the 
project. 
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Figure  8:  Technical  Management  Process  Resource  Allocation 
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Figure  9:  Technical  Process  Resource  Allocation 
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One  concept  not  captured  in  evaluating  the  different  processes  was  iteration.  Most 
of  the  texts  cited  process  iteration  as  a  key  strategy  in  product  development.  Once  a  design 
is  created,  it  is  presented  to  the  stakeholders  for  feedback  and  refinement.  This  could 
happen  multiple  times  but  the  Rapid  Application  Development  team  suggests  at  least  three 
iterations.  The  inability  of  this  evaluation  to  capture  the  importance  of  design  iteration 
could  give  program  managers  a  false  impression  that  a  single  pass  development  strategy 
using  the  above  resource  allocations  will  produce  a  successful  product.  A  more  successful 
strategy  is  to  integrate  the  user  into  the  development  team  providing  constant  feedback  as 
the  product  grows  from  requirements  to  specifications  to  assembly  and  test. 

Key  Activities  Identified  by  AFRL  Rapid  Reaction  Team  Members 

A  purposive  sampling  was  conducted  between  AFRL  Scientist  and  Engineer  (S&E) 
employees  that  have  participated  in  Core  Process  3  (CP3)  projects.  Individual  interviews 
sought  to  establish  a  baseline  of  common  practices  for  project  managers.  The  interviews 
were  conducted  among  engineers  and  program  managers  with  2-6  years  of  experience  in 
AFRL  rapid  development  with  projects  ranging  from  6  months  to  3  years  in  schedule  and 
$500,000  to  $12M  in  budget.  Backgrounds  ranged  from  prior  military  service  to  active  duty 
to  career  civilian  with  positions  in  and  outside  of  AFRL.  Team  sizes  for  their  rapid 
development  projects  ranged  from  3  to  12  people. 

For  each  technical  and  technical  management  process,  a  common  set  of  products 
(i.e.  work  breakdown  structure,  integrated  master  plan,  team  charter,  key  resources  for 
Technical  Planning)  was  evaluated  for  whether  the  subject  fully,  partially  or  did  not 
accomplish  during  their  projects.  (A  full  list  of  the  common  products  can  be  found  in 
Appendix  A.)  The  author  infers  from  this  assessment  that  the  effort  put  to  creating  (or  not 
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creating)  these  products  reflects  a  level  of  importance  the  subject  placed  on  each  technical 
and  technical  management  process.  These  scores  will  be  compared  with  the  literature 
review  scores  and  analyzed  in  Chapter  5. 

Figures  10  and  1 1  show  the  results  from  the  interviews.  The  scores  are  on  the  same 
scale  of  level  of  importance  (1  to  5)  and  reflect  what  project  managers  have  done.  The 
technical  process  scores  at  first  glance  do  not  present  any  “smoking  guns”.  Architecture 
Design  and  Implementation  show  slightly  higher  scores  while  Verification,  Validation  and 
Transition  show  slightly  lower.  Since  rapid  development  projects  are  by  nature  short  on 
schedule,  little  slack  is  built  in  and  thus  an  emphasis  on  doing  more  up  front  is  displayed  in 
the  data.  Subjects  noted  using  engineering  standards  and  monitoring  progress  to  identify 
opportunities  as  ways  to  stay  within  short  timelines. 
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Figure  10:  AFRL  Rapid  Development  Technical  Process  Scores 
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Figure  11:  AFRL  Rapid  Development  Technical  Management  Scores 


The  technical  management  processes,  however,  show  greater  variability.  Decision 
Analysis,  Technical  Planning,  Technical  Assessments  and  Data  Management  all  have 
higher  scores  than  Requirements,  Risk,  Configuration  and  Interface  Management.  During 
interviews,  it  was  commented  that  rapid  development  projects  require  strong  leadership, 
delegation  to  the  most  competent  team  members,  and  emphasizing  sharing  information 
over  documentation  as  successful  strategies  during  rapid  development  projects.  These 
themes  reflect  the  first  three  processes  listed.  The  last  process,  Data  Management,  may 
have  scored  higher  due  to  the  complex  technical  nature  of  the  projects. 

Many  interviewees  had  comments  that  could  not  be  captured  by  the  survey  on  how 
they  do  rapid  development.  The  following  statements  were  from  individuals  and  not 
themes  expressed  by  multiple  people.  One  interviewee  likened  rapid  development  to  a 
“jazz  [band],  not  an  orchestra.”  Another  noted  that  he  would  have  more  “Interim  Program 
Reviews”  with  newer  teams  to  build  up  the  trust  in  the  group  and  cut  back  once  the  team 
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was  performing  at  a  sufficient  level.  His  advice  on  time  management  was  to  “identify  the 
most  critical  risks”  to  the  project  weekly  and  mitigate  during  hour-long  conference  calls 
and  that  rapid  development  “required  strong  leadership.”  The  smaller  risks  were  often  left 
to  individual  team  members,  allowing  the  more  senior  team  members  to  focus  on  the  hard 
problems.  Another  interviewee  said  he  didn't  receive  enough  training  on  risk  management 
when  applied  to  rapid  development.  One  suggested  that  you  don't  use  Microsoft  Project 
and  that  schedules  don't  show  activities  finer  than  one  week. 

One  interviewee  felt  the  current  project  milestones  weren't  chosen  without 
monitoring  a  project,  but  rather  were  held  based  on  the  initial  schedule.  He  felt  that  reviews 
were  being  held  to  catch  problems  and  that  issues  “should  be  caught  before  test  reviews”. 
When  applied  to  software,  he  felt  that  rapid  development  didn't  afford  time  to  check  bugs 
in  code  written  by  geographically  separated  programmers,  that  there  “wasn't  time  for  QA 
[quality  control]. 
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V.  DISCUSSION 


Overview 

This  study  set  out  to  identify  key  processes  within  the  Department  of  Defense 
Systems  Engineering  framework  that  were  of  importance  to  rapid  development  projects 
within  AFRL.  A  content  analysis  of  product  development  literature,  where  industry  strives 
to  be  the  first  to  market  a  quality  product,  was  perfonned  across  six  different  sources 
representing  various  communities  in  product  development-  academia,  consultancy, 
software,  trade  association-  and  across  15  years  of  research.  The  frequency  of  keywords 
derived  from  the  DAG  definitions  of  each  project  was  used  to  infer  an  importance 
emphasized  by  each  of  the  texts.  By  computing  a  relative  score  on  a  scale  of  1  to  5,  certain 
processes  were  shown  to  be  more  important  in  product  development.  Technical  Planning 
stood  out  as  the  most  important  Technical  Management  process  followed  by  Decision 
Analysis,  Technical  Assessment  and  Risk  Management.  Stakeholders  Requirements 
definition  showed  to  be  the  most  important  Technical  Process  with  Architecture  Design, 
Integration  and  Verification 

A  purposive  sampling  of  AFRL  S&E  employees  was  conducted  to  evaluate  the 
current  implementation  of  SE  processes  within  rapid  development  projects.  Interviewees 
were  asked  to  describe  their  approach  to  projects  they  had  participated  in  or  led.  The  author 
evaluated  their  responses  with  a  numerical  score  based  on  how  fully  different  SE  products 
were  created  and  thus  their  importance  inferred.  While  the  technical  processes  showed 
relatively  similar  results,  Architecture  Design  and  Implementation  were  scored  slightly 
higher.  This  has  been  attributed  to  the  short  schedules  of  rapid  development  forcing  more 
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emphasis  “up  front”.  Technical  management  processes  that  scored  higher  included 
Decision  Analysis,  Technical  Planning  and  Assessment,  and  Data  Management.  The  theme 
here  is  that  a  strong  decision  making  framework  (or  strong  leadership)  is  useful  to  keep 
skilled  teams  on  schedule  during  technically  complex  projects. 

When  combined,  the  content  analysis  and  purposive  sampling  offer  an  interesting 
comparison.  Figures  12  and  13  show  both  sets  of  scores  for  the  SE  processes.  First,  we'll 
examine  the  technical  process  scores.  While  both  sets  of  data  agree  that  Architecture 
Design  is  important,  the  literature  does  not  emphasize  Requirements  Analysis  (RA)  nor 
Implementation  to  the  same  degree  that  the  AFRL  S&E"s  place  importance  on  those 
processes.  The  literature  does,  however,  place  a  large  emphasis  on  Stakeholders 
Requirements"  Definition. 


5.00 


4.50 


4.00 


3.50 


3.00 


2.50 


2.00 


1.50 


1.00 


0.50 


0.00 


■ 


SRD  RA  AD  Imp  Int  Ver  Val  Trans 


Figure  12:  Combined  Technical  Process  Scores 
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Since  RA  is  derived  from  the  initial  requirements,  it  may  be  that  the  literature 
leaves  it  to  the  reader  to  perfonn  the  engineering  activities  to  go  from  requirements  to 
design  and  does  not  devote  as  much  time  to  explaining  that  effort.  However,  for  AFRL 
S&E"s  it  is  an  important  piece  of  the  design  process  to  ensure  stakeholders"  concerns  are 
matched  to  perfonnance  specifications.  Implementation  consists  of  designing  and  testing 
the  subsystems  and  components.  This  being  an  internal  process  that  feeds  into  the  final, 
assembled  product,  the  literature  may  place  little  emphasis  compared  to  other  processes. 
AFRL  S&E's  noted  that  constructing  a  prototype  to  show  a  user  early  and  quickly  was  a 
key  step  that  allowed  feedback  to  modify  design  or  requirements. 

The  literature  also  shows  little  emphasis  on  transition  compared  to  AFRL  S&E"s. 
This  could  be  an  assumption  in  the  literature  that  if  you  research,  design,  build  and  test 
successfully  consumers  will  buy  your  product.  In  the  Lab,  product  transition  is  less  of  a 
guarantee  and  subject  to  separate  acquisition  organizations  that  require  advocacy  and 
funding  above  that  required  for  the  rapid  development  project.  The  remainder  process 
scores  are  comparable  to  each  other. 
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Figure  13:  Combined  Technical  Management  Process  Scores 
Next,  we  will  examine  the  technical  management  process  scores.  The  main 
discrepancies  are  in  Technical  Planning  and  Interface  Management.  The  literature  places 
the  most  emphasis  in  determining  the  scope  of  the  technical  effort  and  developing  a 
systems  engineering  plan  to  cover  all  aspects  of  a  project.  However,  many  of  the 
interviewees  attested  that  iterating  on  a  design  with  feedback  from  the  user  was  more 
important  than  developing  a  “fire-proof’  plan.  Interface  Management  was  emphasized 
more  among  AFRL  S&E"sthan  in  the  literature.  This  could  be  due  to  the  integrated  nature 
of  defense  products  especially  with  sensor  technologies  that  are  designed  to  push 
information  and  intelligence  products  across  an  enterprise  of  users.  The  literature  is  either 
not  concerned  primarily  with  products  integrated  with  external  interfaces  such  as  designing 
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a  portable  CD  player  or  assumes  that  the  external  interfaces  exist  and  are  well  defined  like 
the  USB  ports  on  your  personal  computer  and  thus  assigns  it  relatively  little  importance. 

To  summarize  the  technical  process  scores,  the  literature  and  AFRL  S&Es  agreed 
to  the  general  principle  of  “up-front  and  early”  when  conducting  rapid  development.  The 
literature  emphasized  Stakeholders  Requirements  Development  and  Architecture  Design. 
The  S&Es  were  more  unifonn  in  their  results  and  agreed  on  the  importance  of  Architecture 
Design  but  also  emphasized  Implementation.  The  technical  management  processes  were 
also  generally  similar,  but  the  literature  showed  Technical  Planning  was  of  stronger 
importance  and  Interface  Management  of  lesser  importance  when  compared  to  the  AFRL 
S&E  scores. 

A  possible  explanation  for  differences  in  both  analyses  is  the  “practitioner  vs. 
pundit”  effect.  With  respect  to  the  “pundits”,  the  content  analysis  of  the  literature  has 
shown  a  strong  preference  for  one  process  over  another,  in  this  case  Tech  Planning  vs 
Interface  Management.  The  authors  may  be  assuming  a  level  of  understanding  within  their 
intended  audience  that  masks  the  relative  importance  of  each  process.  They  could  also 
overemphasize  processes  that  either  were  ignored  in  the  past  or  were  executed  poorly. 

From  the  point  of  view  of  the  “practitioner”  there  may  be  a  stronger  emphasis  on  the 
processes  that  are  requirements  due  to  policy  or  practicality.  Most  of  the  technical  process 
scores  cluster  around  2.75,  with  a  score  of  3  meaning  the  process  was  “important”  vice 
“very  important”  or  “not  important”  and  the  activities  within  the  process  were  neither  fully 
implemented  nor  fully  ignored.  In  this  study,  the  literature  deems  Implementation  as  “not 
important”.  This  contrasts  with  the  AFRL  engineers  which  score  it  as  “important”.  In 
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reality,  a  project  must  implement  the  design,  otherwise  there  would  be  no  product  to  test  or 
deliver. 

Conclusions 

This  study  set  out  to  detennine  if  there  were  key  Systems  Engineering  Processes 
emphasized  by  product  development  literature  that  could  be  implemented  within  AFRL 
rapid  development  projects.  From  the  literature,  Stakeholders  Requirements  Definition, 
Architecture  Design  and  Technical  Planning  were  strongly  emphasized  when  compared  to 
the  other  processes.  This  agrees  with  the  anecdotal  lesson  learned  “plan  up  front  and  early”. 
While  interviewees  agreed  that  up-front  technical  planning  was  important  to  maintaining 
short  schedules,  progress  in  delivering  a  prototype  iterating  the  design  based  on  user 
feedback  was  as  important.  Based  on  these  results,  project  managers  and  chief  engineers 
participating  in  future  AFRL  CP3  and  other  rapid  development  projects  should  focus  on 
these  processes  early  on  in  the  projects"  lifecycle.  Senior  leaders  should  encourage  training 
in  developing  project  requirements,  architectures  and  holding  meaningful  reviews. 
Recommendations  and  Areas  of  Future  Research 

The  framework  developed  in  this  study  could  serve  as  a  guide  for  program 
managers  of  rapid  development  projects.  AFRL"s  Core  Process  3  teams  could  be  made 
aware  of  the  findings  codified  by  modifying  the  current  AFRL  instruction  for  CP3  or  as  an 
accompanying  AFRL  Manual. 

The  outcome  of  the  importance  of  the  SE  processes  was  highly  dependent  on  the 
materials  chosen.  The  methodology  can  be  implemented  further  by  including  more  product 
development  literature  or  by  focusing  on  a  particular  field  (i.e.  software  development)  and 
comparing  to  case  studies  within  that  field. 
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This  research  was  conducted  to  follow  up  previous  studies  of  rapid  development 
within  AFRL  and  the  AFIT  theses  of  Capt  David  Solomon  and  Majors  Behm,  Pitzer  and 
White  should  also  be  consulted  for  additional  topics. 
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APPENDIX  A:  SE  PRODUCTS  SURVEY 


SME  # _ 

Interview  Questions 

Background 

Average  funding  of  project?: _  Average  schedule0 _ Number  of  projects? 

Tell  me  about  how  you  approach  SE  in  rapid  reaction 
(Walk  through  16  disciplines) 


Technical  Management  Processes 

Technical  Processes 

Decision  Analysis 

Stakeholders  Reauirements  Definition 

Technical  Planning 

Reauirements  Analysis 

Technical  Assessment 

Architectural  Design 

Reauirements  Management 

Implementation 

Risk  Management 

Integration 

Configuration  Management 

Verification 

Technical  Data  Management 

Validation 

Interface  Management 

Transition 
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SME# _ 

Technical  Processes 


Stakeholders  Requirements  Definition  Fully 

How  do  you  flush  out  system  constraints?  q 

Do  you  craft  successful  validation  criteria? 

Do  you  develop  a  concept  document  or  OV-1? 

Requirements  Analysis 

Are  system  functional  requirements  created? 

Are  system  perfomiance  requirements  derived?  0 

Do  you  consider  architectural  constraints?  G 

Do  you  have  a  verification  strategy  for  the  end  G 

product? 

Architecture  Design 

Is  an  architecture  baseline  established  (DoDAF)? 

Are  system  element  descriptions  crafted?  0 

Do  you  have  interface  requirements'1  0 

Do  you  create  a  system  integration  plan?  □ 

Implementation 

Do  you  have  implementation  constraints? 

Is  there  an  implementation  strategy'  to  deal  with  the  □ 
constraints? 

Is  each  planned  subsystem  built?  - 

Do  you  provide  initial  operator  - 

training  evaluation? 

Integration 

Do  you  use  simulations?  - 

Do  you  write  component  test  reports?  J 

Are  there  problem  resolution  reports0 
Do  you  have  interface  control  documents  during 
integration? 

Verification 

Do  you  write  a  test  plan  test  cards?  0 

Do  you  write  test  reports  for  the  whole  system  and  - 
compare  to  the  requirements? 

Do  you  document  corrective  actions? 

Validation 

Do  you  participate  in  a  demo  or  exercise? 

Is  there  an  independent  evaluation0  0 

Is  there  an  Operational  Assessment  in  the  field?  Q 

Transition 

Do  you  provide  a  fielded  system?  0 

Do  you  test  for  interoperability  certifications?  □ 

Is  there  a  formal  safety  evaluation?  □ 

Do  you  write  a  transition  plan? 

Are  the  -ilities  considered0 


Category 

Partially  Did  Not 

□  0 

0  □ 

0  □ 

□  D 

□  0 

0  □ 

n  d 


G 


U 

p 


□ 

0 
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□  aaci  a  a  a  a  nan  an 


SME  # _ 

Technical  Management  Processes 

Decision  Analysis  Fully  Partially 

Do  you  define  and  evaluate  alternatives?  Q  0 

Is  there  relevant  stakeholder  buy-in?  □  n 

Define  strategy  and  success  criteria  q  □ 

Decision  rational  documents?  q  0 

Lessons  Learned?  - 

Technical  Planning 

Work  Breakdown  Structure  -  - 

Roles  and  Responsibilities.  Team  Charter 

Systems  Engineering  Plan  n  n 

Chief  Engineer  and  Key  Resources  _  _ 

Integrated  Master  Plan  _  _ 

Technical  Assessments  ~  ~ 

System  Functional  Review  y  □ 

Preliminary  Design  Review  □  Q 

Critical  Design  Review  D  Q 

Test  Readiness  Review  0  0 

System  Verification  Review 

Interim  Program  Reviews  u  E 

Requirements  Management 

Project  performance  measures  J  -1 

Traceability  matrix  Impact  assessments 
Change  request  documentation 
Risk  Management 

Risk  matrix  0  0 

Risk  tracking  0  0 

Risk  priority  0  □ 

Risk  mitigation  plan  n  0 

Configuration  Management 

Configuration  baseline  □  n 

Configuration  Control  Document  Board' Charter  -  - 

Documented  impact  of  change  request 

Functional  Configuration  Audit  -  j-, 

Physical  Configuration  Audit  -  — 

Data  Management 

Common  managed  data  storage 

Data  rights,  manuals,  documentation  0  0 

Security  classification  guide  0  Q 

Interface  Management 

ICWG  documented  interface  specs  0  D 

Interface  compliance  matrix  q  y 


Did  Not 

0 

□ 

□ 

0 

o 

0 

□ 

□ 

□ 

□ 
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□  □acicm  □  □  □  □  □  □  □  □□□!=)□  ana  □□ 


APPENDIX  B:  LITERATURE  SCORES 


Raw  Scores  (frequency  count) 


Stakeholders 

Requirements 

Definition 

Requirements 

Analysis 

Arch  Design 

Implementation 

Integration 

Verification 

Validation 

Transition 

Wheelright  and  Clark 

12 

4 

8 

4 

7 

6 

6 

0 

Cooper/  Stage  Gates 

9 

1 

3 

1 

2 

4 

3 

2 

Smith  and  Reinertsen 

4 

2 

9 

0 

3 

1 

0 

1 

PLM 

3 

1 

3 

0 

4 

2 

1 

0 

RAD-  Total 

6 

2 

2 

0 

2 

4 

2 

1 

RAD-  Standard,  1995 

3 

1 

1 

0 

1 

3 

1 

1 

RAD-  Implemented,  2000 

3 

1 

1 

0 

1 

1 

1 

0 

PDMA 

12 

2 

5 

1 

2 

2 

2 

3 

Handbook  2, 2005 

11 

1 

4 

0 

1 

1 

1 

2 

Tool  book  3,  2007 

1 

1 

1 

1 

1 

1 

1 

1 

Decision  Analysis 

Technical 

Planning 

Technical 

Assessment 

Requirements 

Management 

Risk  Mgmt 

Config  Mgmt 

Tech  Data  Mgmt 

Interface  Mgmt 

Wheelright  and  Clark 

5 

10 

3 

1 

2 

0 

0 

0 

Cooper/  Stage  Gates 

5 

6 

4 

0 

4 

0 

0 

0 

Smith  and  Reinertsen 

3 

7 

2 

2 

7 

2 

2 

1 

PLM 

1 

3 

0 

1 

0 

3 

4 

0 

RAD-  Total 

1 

3 

1 

2 

1 

2 

2 

0 

RAD-  Standard,  1995 

1 

2 

0 

0 

1 

2 

0 

0 

RAD-  Implemented,  2000 

0 

1 

1 

2 

0 

0 

2 

0 

PDMA 

7 

7 

8 

1 

5 

1 

1 

1 

Handbook  2, 2005 

6 

6 

7 

0 

4 

0 

0 

0 

Tool  book  3,  2007 

1 

1 

1 

1 

1 

1 

1 

1 

Normalized  Scores 


Technical  Processes 

SRD 

RA 

AD 

Imp 

Int 

Ver 

Val 

Trans 

Wheelright  and  Clark 

5.00 

1.67 

3.33 

1.67 

2.92 

2.50 

2.50 

0.00 

Cooper/  Stage  Gates 

5.00 

0.56 

1.67 

0.56 

1.11 

2.22 

1.67 

1.11 

Smith  and  Reinertsen 

2.22 

1.11 

5.00 

0.00 

1.67 

0.56 

0.00 

0.56 

PLM 

3.75 

1.25 

3.75 

0.00 

5.00 

2.50 

1.25 

0.00 

RAD 

5.00 

1.67 

1.67 

0.00 

1.67 

3.33 

1.67 

0.83 

PDMA 

5.00 

0.34 

0.86 

0.17 

0.34 

0.34 

0.34 

0.52 

Average 

4.33 

1.10 

2.71 

0.40 

2.12 

1.91 

1.24 

0.50 

St  Dev 

1.15 

0.55 

1.57 

0.66 

1.64 

1.19 

0.93 

0.44 

Percent  of  Total 

30.3% 

7.7% 

19.0% 

2.8% 

14.8% 

13.3% 

8.7% 

3.5% 

Percent  St  Dev 

22.93% 

11.07% 

31.38% 

13.15% 

32.87% 

23.84% 

18.53% 

8.89% 

Technical 

Management 

Processes 

DA 

TP 

TA 

Req  Mgmt 

Risk  Mgmt 

Config  Mgmt 

Data  Mgmt 

Int  Mgmt 

Wheelright  and  Clark 

2.50 

5.00 

1.50 

0.50 

1.00 

0.00 

0.00 

0.00 

Cooper/  Stage  Gates 

4.17 

5.00 

3.33 

0.00 

3.33 

0.00 

0.00 

0.00 

Smith  and  Reinertsen 

2.14 

5.00 

1.43 

1.43 

5.00 

1.43 

1.43 

0.71 

PLM 

1.25 

3.75 

0.00 

1.25 

0.00 

3.75 

5.00 

0.00 

RAD 

1.67 

5.00 

1.67 

3.33 

1.67 

3.33 

3.33 

0.00 

PDMA 

4.38 

4.38 

5.00 

0.63 

3.13 

0.63 

0.63 

0.63 

AVERAGE 

2.68 

4.69 

2.15 

1.19 

2.35 

1.52 

1.73 

0.22 

St  Dev 

1.30 

0.52 

1.75 

1.17 

1.81 

1.65 

2.03 

0.35 

Percent  of  Total 

16.2% 

28.3% 

13.0% 

7.2% 

14.2% 

9.2% 

10.5% 

1.3% 

Percent  St  Dev 

26.04% 

10.46% 

35.01% 

23.44% 

36.22% 

33.10% 

40.56% 

6.94% 

Totals 

47 

25 

20 

14 

19 

11 

8 

29 

21 

8 


21 

19 

26 

12 

12 

6 

6 

31 

23 

8 
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APPENDIX  C:  SME  SCORES 


Background  Data 


ID 

Yrs  RD  Exp 

Min  Sched 
(years) 

Max  Sched 
(years) 

Min  Budget 
($’000) 

Max  Budget 
($'000) 

Min 

Team  Size 

Max 

Team  Size 

1 

6 

0.5 

3 

500 

3000 

1 

10 

2 

5 

0.5 

2 

500 

1000 

3 

12 

3 

2 

2 

2 

70000 

70000 

90 

90 

4 

2 

0.5 

1.5 

500 

2000 

6 

12 

5 

3 

1 

1.5 

600 

1000 

3 

6 

6 

2 

1 

2 

20000 

100000 

20 

50 

7 

3 

0.5 

2 

500 

2000 

5 

12 

Technical  Process  Scores 


SRD 

RA 

AD 

Imp 

Int 

Ver 

Val 

Trans 

1 

3.40 

3.50 

2.50 

3.00 

3.50 

1.00 

1.00 

2.33 

2 

4.00 

4.00 

4.33 

4.75 

2.00 

2.00 

1.67 

1.00 

3 

3.00 

3.00 

2.00 

3.00 

2.50 

3.00 

5.00 

3.00 

4 

1.50 

2.33 

3.67 

3.00 

3.50 

3.00 

3.00 

3.00 

5 

3.00 

3.00 

5.00 

3.00 

2.00 

2.00 

2.33 

3.00 

6 

5.00 

5.00 

4.00 

5.00 

4.50 

5.00 

4.00 

4.20 

7 

1.67 

1.50 

4.00 

3.67 

3.67 

3.67 

2.00 

1.00 

Technical  Management  Scores 


ID 

DA 

TP 

TA 

Req 

Mgmt 

Risk 

Mgmt 

Config 

Mgmt 

Data 

Mgmt 

Int 

Mgmt 

1 

3.25 

2.00 

1.00 

1.50 

1.00 

1.00 

2.00 

1.00 

2 

3.00 

4.40 

2.33 

1.67 

2.00 

1.00 

4.00 

3.00 

3 

2.60 

2.60 

2.20 

2.33 

2.50 

2.20 

4.00 

2.00 

4 

4.00 

3.40 

3.00 

1.67 

1.00 

1.50 

1.00 

1.00 

5 

3.50 

2.50 

2.67 

3.00 

3.00 

1.00 

1.00 

1.00 

6 

3.00 

4.33 

5.00 

2.33 

5.00 

3.00 

4.00 

3.00 

7 

1.80 

1.00 

2.60 

1.00 

1.00 

2.20 

3.00 

5.00 
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